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DETAILED ACTION 

1 . This Office Action is responsive to communication filed March 14, 2007. 

2. Claims 20 - 39 are pending in this Office Action. 

Response to Arguments 

(a) Applicant's remarks states that ^Hhe MediaNet publication was published on 
November 6, 200 by three co-authors^ two of whom are co-inventors of the present pending 
application and claims (page 7, paragraph 6). 

Examiner respectfully disagrees with the applicant. The co-authors of MediaNet 
publication are Ana B. Benittez, John R. Smith and Shih-Fu Chang while the inventors 
of the presently pending application and claims are Yuan-Chi Chang, Chung-Sheng Li, 
Apostol Ivanov and John R. Smith. Contrary to applicant's assertion, only one co- 
inventor (John R. Smith) of the presently pending application is a co-author of MediaNet 
publication. 

Joint inventors must apply for a patent jointly and each must make the required 
oath or declaration: neither of them alone, nor less than the entire number, can 
apply for a patent for an invention invented by them jointly, except as provided in 
37 CFR § 1.47. Therefore, applicant's assertion is not an evidence to constitute a 
reduction to practice as required by the last Office Action. 
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(b) U.S. Provisional Patent Application Serial No. 60/246,052 (the Provisional 
Application) was filed on November 6, 2000, which is the publication date of the Benitez 
(MediaNet) publication (page 7, paragraph 6). 

Examiner submits that the presently pending application does not claim a priority 
to Provisional Application No. 60/246,052. ^ 

If applicant desires to claim the benefit of a prior-filed application under 35 U.S.C. 
1 19(e), a specific reference to the prior-filed application in compliance with 37 CFR 
1 .78(a) must be included in the first sentence(s) of the specification following the title or 
in an application data sheet. For benefit claims under 35 U.S.C. 120, 121 or 365(c), the 
reference must include the relationship (i.e., continuation, divisional, or continuation-in- 
part) of the applications. Therefore, Provisional Application No. 60/246,052 is not an 
evidence to constitute a reduction to practice as required by the last Office Action. 

(c) Examiner should recognize that the filins of the Provisional Application in the 
United States on the same date of MediaNet publication is unequivocal proof that shows a 
reduction to practice of the invention in this country before the date of the printed MediaNet 
publication (page 8, paragraph 2). 

The mere assertion that MediaNet publication and Provisional Application No. 
60/246,052 are both published and filed respectively on the same date (November 6, 
2000) is not an evidence to constitute a reduction to practice as required by the last 
Office Action because neither MediaNet publication nor Provisional Application No. 
60/246,052 is claimed by presently pending application as a priority date. 
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3. In view of the above response, examiner contends that the applicant has not 
provided evidence to support the reduction to practice before the date of printed 
MediaNet publication; therefore MediaNet reference is still valid. 

Claim Rejections - 35 USC §103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the jsubject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S.C. 103(a). 

5. Claims 20, 27 and 34 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Non-Patent Document: "Supporting Ranked Boolean Similarity Queries in MARS", 
IEEE Trans, on Knowledge and date Engineering, 10, Nov - Dec. 1998, Author: Ortega 
et al (hereinafter "Ortega") in view of Non-Patent Document: "MediaNet: A multimedia 
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Information Network for Knowledge Representation", In Conference on Internet 
Multimedia Management Systems. Vol. 4210 (numbered Pages 1 - 12), Boston, MA, 
Nov. 2000, 1 ST/SPIE.OO, Author: Benitez et al (hereinafter "Benitez"). 

Regarding claims 20, 27 and 34, Ortega teaches a method for processing 
multimedia data in a computer system, comprising: 

receiving as input a high-level concept describing data to be accessed (see section 
1.2, page 4, paragraph 1, "A Boolean retrieval model (adapted for retrieval over images) 

is used to interpret the query being able to support such conceptual queries is 

critical for the versatility of large image databases.", and section 2, "In this section, we 
briefly describe the image features used . . . Other image features are available , 
however we restrict ourselves to queries involving only to the above features in this 
paper."); 

translating the high-level concept into a low-level query by using stored concept 
constructs which are defined using features derived from a plurality of application 
domains (see section 1.2, page 4, paragraph 1, "A Boolean retrieval model (adapted for 
retrieval over images) is used to interpret the query ..... . being able to support such 

conceptual queries is critical for the versatility of large image databases.", and section 2, 
"In this section, we briefly describe the image features used . . . Other image features 
are available , however we restrict ourselves to queries involving only to the above 
features in this paper."). 
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Ortega does not explicitly teach transferring the low-level query to one or more 
search engines and concept repository. 

Benitez teaches transferring the low-level query to one or more search engines to 
access information using the low-level query (see section 3, page 6, "Typical content- 
based retrieval systems indexed ... or the value for any low-level features in the image 

database" and section 4.2, page 9, "At this point and the results integrated into a 

unique list as described for visual queries"); and 

a concept repository for storing and accessing the concept constructs (see Fig. 4 and 
section 4.1, page 7, "For each application, the list of concepts and relationships in the 
MediaNet knowledge based should be representative of the content in the database and 
the goal of the application task."). 

It would have been obvious to one of ordinary skill in the data processing art at the 
time of the present invention to combine teaching of the cited references because 
Benitez's teaching of "transferring the low-level query to one or more search engines to 
access information using the low-level query" and "concept repository" would have 
allowed Ortega's system to integrate both conceptual and perceptual representations of 
knowledge to impact a broad range of applications that deal with multimedia content at 
semantic and perceptual levels. This improves the performance of multimedia retrieval 
applications by using query expansion, refinement and translation across multiple 
content modalities as suggested by Benitez (see Abstract). 
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6. Claims 21 - 26, 28 - 33, and 35 - 39 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Ortega in view of Benitez and further in view of Non-Patent 
Document: "CAMEL: Concept Annotated image Libraries", In Storage and Retrieval for 
Image and Video Database, San Jose, CA, Jan. 2001. SPIE (numbered pages 1-12), 
Author: Natsev et al (hereinafter "Natsev"). 

Regarding claims 21, 28 and 35, Ortega teaches storing matching algorithms in a 
matching algorithm library module (see section 4.8, page 20 - section 4.8.3, page 25) 
and Benitez teaches storing the concept constructs in a concept library module (see 
Fig.4 and section 4.1 , page 7, "For each application, the list of concepts and 
relationships in the MediaNet knowledge based should be representative of the content 
in the database and the goal of the application task."). 

Ortega or Benitez does not explicitly teach storing the features in a feature library 
module; and storing constraints in a constraint library module. 

However, Natsev further teaches storing the concept constructs in a concept library 
module (see section 1.2, "The concept cataloguing, or learning, phase is used to define 
visual concepts and build a concept library. Concepts are defined .... The concept 
library is module for a persistent storage of concepts"); 

storing the features in a feature library module (see fig. 3); and 

storing constraints in a constraint library module (see section 6, "Another 
improvement that we are considering is the introduction of spatial constraints in the 
query engine. For example, . . . , the corresponding document is ranked higher"). 
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It would have been obvious to one of ordinary skill in the data processing art at the 
time of the present invention to combine teaching of the cited references because 
Natsev^s teaching of storing the concept constructs in a concept library module, storing 
the features in a feature library module and storing constraints in a constraint library 
module would have presented a new direction for improving current image query 
system, namely focusing the query specification part as gateway to better performance 
and usability (see Natsev section 6) for the combination Ortega and Benitez's system. 

Regarding claims 22, 29 and 36, Natsev teaches interfacing the library modules 
to the application domains (see fig. 3 and section 4, "Fig. 3 illustrates the architecture of 
the . . . , as well as the Concept Library module"). 

Regarding claims 23, 30 and 37, Natsev teaches building a concept construct 
(see fig.1 and section 1.2, The concept cataloguing, or learning, phase is used to define 
visual concepts and build concept library. Concepts are defined . . . and to associate it 
with the given concept"). 

Regarding claims 24 and 31 , Natsev teaches a method as defined in Claim 23, 
wherein the step of building a concept construct comprise combining one or more of the 
features with see section 1.2, The concept cataloguing, or learning, phase is used to 
define visual concepts and build concept library) 
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zero or more of the stored concept (see Fig.1 and section 1.2, "The concept 
library is a module for persistent storage of concept"), and 

zero or more of the constraints (see section 6, "Another improvement that we are 
considering is the introduction of spatial constraint in query engine"). 

Regarding claims 25 and 32, Ortega teaches a concept construct is represented 
using a hierarchical fuzzy graph data tree-structure comprising: nodes that correspond 
to child-concepts and a subset of the features; aggregation edges that correspond to 
parent-child relationships; and association edges that correspond to inter-sibling 
constraints (see sections 4.1 and 4.2, pages 10 and 11). 

Regarding claims 26 and 33, Natsev teaches a method as defined in Claim 20, 
wherein the features are user defined (see section 2, "One of the most commonly used 
image features is color histogram, . . . places the burden on the user to specify weights 
for the different features"). 

Regarding claim 38, Natsev teaches a system as defined in Claim 34, wherein 
the translation engine further comprises an interpreter that translates the high level 
concept (see fig.1). 
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Regarding claim 39, Natsev teaches a system as defined in Claim 34, further 
comprises a search engine (see section 2, "the high-level approach that . . . . , there are 
some considerations that make certain query engines more suitable than others". 

Conclusion 

7. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from' the mailing date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Fred I. Ehichioya whose telephone number is 571-272- 
4034. The examiner can normally be reached on M - F 8:00 AM to 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John E. Breene can be reached on 571-272-4107. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



Fred I. Ehichioya 
Patent Examiner 
Art Unit 2162 
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